Augmented reality displays with active alignment and corresponding methods

ABSTRACT

Binocular augmented reality display devices and corresponding methods allow alignment calibration to be performed by an end user. According to one approach, a camera is positioned to have a field of view which includes simultaneously part of a projected image from the left-eye display and part of a projected image from the right-eye display. By projecting via each display at least part of a calibration image and identifying within the camera-sampled image right-field and left-field alignment features, an alignment correction can be derived. Alternative approaches employ correlation of images sampled by forward-looking cameras rigidly associated with the respective right-eye and left-eye display units, or require a user to input a manual adjustment for aligning transversely-swapped camera images with the real world view.

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to augmented reality displays and, inparticular, it concerns binocular augmented reality displays witharrangements for adjusting alignment of the left-eye and right-eyedisplays of a binocular augmented reality display, and correspondingalignment methods.

Augmented reality spectacles must be aligned accurately in order toprovide an effective binocular observation experience of the augmentedimage, and even relatively small misalignment may risk causing eyestrain or headaches. Conventional approaches typically involve mountingthe left-eye and right-eye displays on a mechanically rigid commonsupport structure, illustrated in FIG. 1A, to achieve preliminaryalignment and a fixed relative position of the displays. Final finealignment is achieved by electronic shift of the image, as illustratedschematically in FIG. 1B, which shows an image generating matrix 30(i.e., the physical extremities of the display field of view), and atransformed projected image 32 according to a calibration matrix,typically programmed into firmware associated with each display, toachieve correct alignment between the displays. The margins between 30and 32 are designed into the system to accommodate any transformationrequired to correct misalignment within predefined limits.

An exemplary alignment process according to this approach is illustratedherein with reference to FIGS. 1A-2. The electronic alignment parametersare generated by placing the spectacles in front of two co-alignedcameras and comparing the orientation of the augmented images generatedby the two projectors. The derived calibration data is introduced to thetransformation firmware of the image projectors. Alternatively, themechanical alignment of the optical system can be accurate to within therequired optical accuracy. The above alignment process requires adedicated optical alignment bench, and is only suitable forimplementation in a production facility.

There is a need to implement augmented reality spectacles in alightweight and compact form factor in order to make the technology moresuitable for the consumer market. Lightweight implementations, however,often lack sufficient mechanical rigidity to ensure invariant alignmentof the two displays over time, instead being subject to variations dueto thermal variations and other mechanical or environmental influences.

Additionally, the inter-pupillary distance (IPD, distance between theeyes) can vary by up to 15 millimeters for different people. As aresult, if the two projectors are connected rigidly, each of theeye-boxes (i.e., the illumination area of each projector where the eyepupil is expected to be, shown as region 10 in FIG. 1A) must be wider by15/2=7.5 mm for each eye in order to accommodate every possible userhaving any IPD within the defined margin. The large eye-box dictatesbulkier and more expensive optics. If a mechanism is provided for IPDadjustment, this typically introduces additional uncertainty into thealignment between the two displays, rendering any pre-calibratedalignment correction unreliable.

SUMMARY OF THE INVENTION

The present invention is a binocular augmented reality display with anarrangement for adjusting alignment of the left-eye and right-eyedisplays of a binocular augmented reality display, and a correspondingalignment method.

According to the teachings of an embodiment of the present inventionthere is provided, a method for deriving an alignment correction betweena right-eye display and a left-eye display of a binocular augmentedreality display device, the method comprising the steps of: (a)positioning a camera having a field of view so that the camera field ofview includes simultaneously part of a projected image from the left-eyedisplay and part of a projected image from the right-eye display; (b)projecting via each of the right-eye display and left-eye display atleast part of a calibration image including at least one right-fieldalignment feature and at least one left-field alignment feature; (c)employing the camera to sample an image; (d) identifying within theimage the right-field alignment feature and the left-field alignmentfeature; and (e) deriving from a position within the image of theright-field alignment feature and the left-field alignment feature analignment correction between the right-eye display and the left-eyedisplay of the augmented reality display device.

According to a further feature of an embodiment of the presentinvention, the camera is positioned on the viewing side of the augmentedreality display device, such that the image includes the right-fieldalignment feature viewed via the right-eye display and the left-fieldalignment feature viewed via the left-eye display.

According to a further feature of an embodiment of the presentinvention, the projected calibration image is displayed with an apparentfocal distance, and wherein the camera is focused at the apparent focaldistance.

According to a further feature of an embodiment of the presentinvention, the camera is positioned on an opposite side from the viewingside of the augmented reality display device so that the camera capturesan outwardly reflected portion of image illumination from each of theright-eye display and the left-eye display, and such that the imageincludes the left-field alignment feature viewed via the right-eyedisplay and the right-field alignment feature viewed via the left-eyedisplay.

According to a further feature of an embodiment of the presentinvention, the camera is a hand-held camera, the method furthercomprising displaying via the right-eye display and/or the left-eyedisplay at least one indication to a user to assist in correctpositioning of the camera.

According to a further feature of an embodiment of the presentinvention: (a) features associated with the binocular augmented realitydisplay device sufficient to define at least three fiducial points areidentified within the image; and (b) a position of the camera isdetermined relative to the at least three fiducial points.

According to a further feature of an embodiment of the presentinvention, the positioning includes directing the camera towards amirror so that the reflected field of view includes simultaneously partof a projected image from the left-eye display and part of a projectedimage from the right-eye display.

According to a further feature of an embodiment of the presentinvention, the camera is a camera of a mobile device integrated with ascreen, the method further comprising displaying via the screen at leastone indication to a user to assist in correct positioning of the camera.

According to a further feature of an embodiment of the presentinvention, an alignment correction to the augmented reality displaydevice is implemented based on the derived alignment correction.

There is also provided according to the teachings of an embodiment ofthe present invention, a method for stereoscopic alignment correctionbetween a right-eye display and a left-eye display of a binocularaugmented reality display device, the method comprising the steps of:(a) providing an augmented reality device comprising: (i) a right-eyedisplay unit comprising a first augmented reality display rigidlyintegrated with a forward-looking first camera, (ii) a left-eye displayunit comprising a second augmented reality display rigidly integratedwith a forward-looking second camera, and (iii) a support structureinterconnecting between the right-eye display unit and the left-sidedisplay unit; (b) providing a first alignment mapping between the firstcamera and the first augmented reality display and a second alignmentmapping between the second camera and the second augmented realitydisplay; (c) sampling at least one image from the first camera; (d)sampling at least one image from the second camera; (e) co-processingthe images from the first and second cameras to derive an inter-cameramapping indicative of a relative orientation between the first cameraand the second camera; (f) combining the inter-camera mapping with thefirst alignment mapping and the second alignment mapping to derive aninter-display alignment mapping indicative of a relative orientation ofthe first augmented reality display and the second augmented realitydisplay; and (g) implementing an alignment correction to the augmentedreality display device based on the inter-display alignment mapping.

According to a further feature of an embodiment of the presentinvention, the at least one image from the first camera and the secondcamera are sampled for a distant scene.

According to a further feature of an embodiment of the presentinvention, the at least one image from the first camera and the secondcamera are multiple images, and wherein the co-processing includesderiving a three-dimensional model of at least part of a scene includedin the multiple images.

There is also provided according to the teachings of an embodiment ofthe present invention, a method for stereoscopic alignment correctionbetween a right-eye display and a left-eye display of a binocularaugmented reality display device, the method comprising the steps of:(a) providing an augmented reality device comprising a right-eyeaugmented reality display, a left-eye augmented reality display, a rightcamera spatially associated with the right-eye augmented realitydisplay, and a left camera spatially associated with the left-eyeaugmented reality display; (b) performing a first cross-registrationprocess comprising: (i) obtaining at least one image of a scene sampledby the right camera, (ii) displaying via the left-eye augmented realitydisplay at least one alignment feature derived from the at least oneimage sampled by the right camera, (iii) receiving an input from theuser indicative of an alignment offset between the at least onealignment feature and a corresponding directly-viewed feature of thescene, and (iv) correcting a position of display of the at least onealignment feature according to the user input until the at least onealignment feature is aligned with the corresponding directly-viewedfeature of the scene; (c) performing a second cross-registration processcomprising: (i) obtaining at least one image of a scene sampled by theleft camera, (ii) displaying via the right-eye augmented reality displayat least one alignment feature derived from the at least one imagesampled by the left camera, (iii) receiving an input from the userindicative of an alignment offset between the at least one alignmentfeature and a corresponding directly-viewed feature of the scene, and(iv) correcting a position of display of the at least one alignmentfeature according to the user input until the at least one alignmentfeature is aligned with the corresponding directly-viewed feature of thescene; and (d) implementing an alignment correction to the augmentedreality display device based on the user inputs.

According to a further feature of an embodiment of the presentinvention, the at least one alignment feature for each of thecross-registration processes is at least part of the sampled image.

According to a further feature of an embodiment of the presentinvention, the at least one alignment feature for each of thecross-registration processes is a location marker corresponding to afeature detected in the sampled image.

According to a further feature of an embodiment of the presentinvention, an estimated distance to an object in the sampled image isobtained, the estimated distance being employed to implement thealignment correction.

According to a further feature of an embodiment of the presentinvention, the right camera is rigidly mounted relative to the right-eyeaugmented reality display, and wherein the left camera is rigidlymounted relative to the left-eye display, the alignment correction beingimplemented using relative alignment data for the right camera relativeto the right-eye augmented reality display and relative alignment datafor the left camera relative to the left-eye augmented reality display.

According to a further feature of an embodiment of the presentinvention, at least one additional registration process is performed toreceive user inputs for correcting an alignment of at least one of theright-eye augmented reality display and the left-eye augmented realitydisplay relative to the corresponding one of the right camera and theleft camera.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1A, described above, is a top view of a binocular augmented realitydisplay according to the prior art;

FIG. 1B is a schematic representation explaining a principle ofelectronic alignment correction for augmented reality displays;

FIG. 2 is a flow diagram illustrating a factory adjustment process forcalibrating an augmented reality display according to the prior art;

FIG. 3 is a schematic front view of a binocular augmented realitydisplay with an arrangement for adjusting IPD, constructed and operativeaccording to an embodiment of the present invention;

FIG. 4 is a schematic side view of the display of FIG. 3 in use;

FIG. 5 is a schematic side view of the device of FIG. 4 during a factorypartial-calibration procedure according to a first implementationoption;

FIG. 6 is a schematic side view of the device of FIG. 4 during a factorypartial-calibration procedure according to a second implementationoption;

FIG. 7 is a schematic representation of a calibration process includingsampling a plurality of images of an object or scene from differentdirections;

FIG. 8 is a flow diagram illustrating a method for alignment calibrationfor the augmented reality display of FIGS. 3 and 4 according to anaspect of the present invention;

FIGS. 9A and 9B are side and front schematic views, respectively, of anaugmented reality display device employing an alternative technique foralignment calibration;

FIG. 9C is a schematic representation of an alignment adjustmentperformed by a user according to this aspect of the present invention;

FIG. 10A is a schematic side view of an augmented reality display deviceduring implementation of an alignment calibration according to a furtheraspect of the present invention;

FIG. 10B is an enlarged schematic side view showing two possiblegeometries of light guiding optical elements for delivering an augmentedreality image to the eye of a user;

FIG. 11A is a schematic top view of the arrangement of FIG. 10A;

FIG. 11B is a schematic top view of a variant implementation of thearrangement of FIG. 10A;

FIG. 11C is a schematic representation of a mobile communications deviceemployed as a camera for the alignment calibration of FIG. 10A;

FIG. 11D is a schematic representation of a calibration image fordisplay via the augmented reality display during performance of analignment calibration according to this aspect of the present invention;

FIG. 11E is a schematic representation of an image sampled by a cameraduring performance of an alignment calibration according to this aspectof the present invention;

FIG. 11F is a schematic top view of a further variant implementation ofthe arrangement of FIG. 10A; and

FIG. 12 is a flow diagram illustrating a method for alignmentcalibration according to the arrangements of FIGS. 10A, 11A, 11B and11F.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is a binocular augmented reality display with anarrangement for adjusting alignment of the left-eye and right-eyedisplays of a binocular augmented reality display, and correspondingalignment methods.

The principles and operation of devices and methods according to thepresent invention may be better understood with reference to thedrawings and the accompanying description.

By way of introduction, the present invention addresses a range ofsituations in which pre-calibrated alignment between a right-eye displayand a left-eye display of a binocular augmented reality display eitherdoes not exist or cannot be considered reliable. This may be due to theuse of lightweight structural components which cannot ensure invariantrigid alignment of the components over an extended period of time and/orvarying environmental conditions, or may be due to the presence of anadjustment mechanism, particularly an IPD adjustment mechanism, whichmay result in imprecise final alignment of the displays. Presence of anIPD adjustment mechanism is particularly preferred, thereby allowing anaugmented reality display device to accommodate users with differinginter-pupillary distances while reducing the requirements for projectoreye-box size and consequent projector bulk, complexity and cost.However, an IPD adjustment mechanism typically introduces variabilityinto the alignment of the two display projectors.

To address these issues, the present invention provides three groups ofsolutions which allow calibration, or recalibration, of alignment of theright and left eye displays of a binocular augmented reality displaydevice in the end-user's normal working environment, and without theneed for any specialized equipment. Specifically, a first subset ofalignment correction techniques are implemented as an automated, orsemi-automated, alignment process based on correlation of images sampledby bilateral cameras associated with respective left and right eyedisplays. A second subset of alignment correction techniques, alsoutilizing cameras mounted on the device, requires user inputs to aligndisplayed features with corresponding real-world features. Finally, athird subset of alignment correction techniques are applicable withoutreliance on cameras mounted on the device, instead relying upon anexternal camera. Each of these subsets of techniques also preferablycorresponds to a distinct implementation of a binocular augmentedreality device with control components configured to implement thecorresponding technique(s). Each approach will now be described indetail.

Referring now to the drawings, FIG. 3-8 illustrate various aspects of abinocular augmented reality display device, an initial partial alignmentprocess, and a corresponding method for stereoscopic alignmentcorrection between a right-eye display and a left-eye display of abinocular augmented reality display device, all according to a firstapproach of an aspect of the present invention. According to thisapproach, each of the two displays (“projectors”) is rigidly attached toa forward looking camera. A support structure bridging between theeye-projectors is relatively less rigid and/or can be modified andlocked by the user according to his or her personal IPD. The images of ascene received by the cameras are compared and a transformation matrixis derived for the projectors.

Thus, in general terms, there is provided an augmented reality devicethat includes a right-eye display unit having a first augmented realitydisplay rigidly integrated with a forward-looking first camera, and aleft-eye display unit having a second augmented reality display rigidlyintegrated with a forward-looking second camera. The augmented realitydevice also includes a support structure interconnecting between theright-eye display unit and the left-side display unit. According to apreferred aspect of this approach, each display unit is rigid, such thateach camera is in fixed alignment with the corresponding augmentedreality display, and the system is provided with, or derives, analignment mapping between each camera and the corresponding augmentedreality display, typically in the form of a transformation matrix whichmaps the camera alignment to the display, i.e., that would allow displayof the camera image correctly aligned with the real world for a distantscene viewed through the augmented reality display. The supportstructure, on the other hand, is not necessarily assumed to sufficientlyrigid to provide invariant alignment between the left-eye and right-eyedisplay units over a period of time, and in certain particularlypreferred implementations, includes an adjustment mechanism, allowingadjustment of the IPD for different users, which typically results insome variation of angular alignment during adjustment.

An alignment correction is then preferably performed by a processingsystem associated with the augmented reality display device, which maybe an onboard processing system or may be a processing system associatedwith the device via a short-range or long-range communicationconnection. Here and elsewhere in this application, the processingdescribed may be performed by standard processing components, which maybe general purpose hardware configured by suitable software, or ASICs orother dedicated or semi-dedicated hardware, as readily chosen by aperson ordinarily skilled in the art according to what it most suited tothe functions described herein. Furthermore, the processing may beperformed at any location, or combination of locations, including butnot limited to, one or more onboard processor forming part of theaugmented reality display device, a mobile communications device inwired or wireless connection with the AR display device, a serverlocated at a remote location and connected to the AR display device viaa WAN, and a cloud computing virtual machine made up of dynamicallyallocated computing resources. Details of the processing systemimplementations are not necessary for an implementation of the presentinvention, and will therefore not be described herein in further detail.

The alignment correction process according to an aspect of the presentinvention preferably includes:

-   -   i. sampling at least one image from the first camera;    -   ii. sampling at least one image from the second camera;    -   iii. co-processing the images from the first and second cameras        to derive an inter-camera mapping indicative of a relative        orientation between the first camera and the second camera;    -   iv. combining the inter-camera mapping with the first alignment        mapping and the second alignment mapping to derive an        inter-display alignment mapping indicative of a relative        orientation of the first augmented reality display and the        second augmented reality display; and    -   v. implementing an alignment correction to the augmented reality        display device based on the inter-display alignment mapping.

This process will be discussed in more detail below.

FIG. 3 depicts schematically a front view of the system according tothis invention. Optics assemblies 40R and 40L project the image intocorresponding see-through optical elements 42R and 42L, preferablyimplemented as transparent light-guiding optical elements with eitherpartial-reflectors or diffractive optical elements for coupling-out avirtual image onto the right and left eyes of the observer,respectively. Forward facing cameras 44R and 44L are rigidly attached totheir adjacent projectors, while a support structure 46, preferablyimplemented as an adjustable mechanical arrangement, connects the twoprojectors. This mechanical arrangement can preferably be unlocked tochange the distance between the projectors and then locked again beforeuse. This enables IPD adjustment and therefore reduction of projectorsize and complexity. It is understood that accurate parallelism andorientation cannot typically be maintained after unlocking and lockingof arrangement 46.

FIG. 4 shows a schematic side view representation of the left projectorand camera. The light from optics 40L passes through the waveguide 42Land is deflected towards the eye (the method of deflection not beingdepicted, but typically based on a substrate with internal obliquepartially-reflective facets as commercially available from Lumus Ltd.,or on an arrangement of diffractive optical elements). Object 50 or thescenery, are imaged by camera 44L. The same object is imaged by theright camera 44R.

The alignment correction process according to this aspect of theinvention requires determination of an alignment mapping between eachcamera and the corresponding augmented reality display for each of theright-eye and the left-eye display units. The transformation parametersbetween the camera axis and the projector axis are preferably measuredafter camera-projector integration, preferably as part of themanufacture process. Various techniques may be used to determine thealignment mapping. Two options will now be described with reference toFIGS. 5 and 6.

In FIG. 5, an external jig 52 holds firmly co-aligned a projector 54 anda camera 56. The projector and the camera are preferably aligned withtheir optical axes parallel to each other, and most preferably, withsufficient accuracy such that no transformation parameters are neededbetween the two. Projector 54 projects a ‘reference image’ that isreceived by camera 44L. A processing system injects a similar centeredimage to projector 40L which generates a projected image which isreceived via optical element 42L by camera 56. The processing systemcompares the images from 44L and 56 to define the transformationparameters between 40L and 44L. The distance between 44L and 42L(specifically, the eye-box center of this waveguide) is also preferablyrecorded for parallax calculations if needed.

In FIG. 6, two projectors 54U and 54D are rigidly attached (or mayalternatively be implemented as a single projector having a sufficientlylarge aperture) and project a calibration image, typically collimated atinfinity. The image from 54U is received by camera 44L and is “injected”into projector 40L. In this case, camera 56 receives simultaneouslythrough optical element 42L a superposition of the directly viewed imageprojected by 54D and the image projected by projector 40L. Thedifferences between the two images correspond to the transformation databetween projector 40L and camera 44L. Most preferably, an automatedalignment process may adjust alignment of the image generated byprojector 40L until a sharp (precisely overlaid) image is received bycamera 56, although a manually-controlled adjustment process using asuitable graphic user interface (not shown) is also possible. Thisadjustment need not actually be implemented in the device firmware atthis stage, since the final alignment will depend also upon thebinocular alignment. To facilitate manual or automated alignment, thealignment image may be an X crosshair or the like, and for clarity ofdifferentiation during the alignment process, the color of the imagefrom 40L may be changed, or the image may be made to blink. The twovisually-distinguished X crosshairs then need to be brought intoalignment.

If optics on projector 42L generate the virtual image at a finitedistance, then it is preferable that the calibration image andconversion of 54U and 54D also be set to this distance, and the imageprojected from projector 40L be shifted when injected to 42L accordingto parallax between camera 44L and projector 42L, since the distanceparameters are known.

The above alignment processes, illustrated for the left-eye displayunit, are clearly repeated (or performed simultaneously) for theright-eye display unit. The result is a well-defined transformationmatrix which maps the camera alignment to the display for each of thedisplay units.

After using one of the above alignment techniques, during or aftermanufacture, to derive the alignment transformation between eachprojector and its corresponding camera, the cameras can then be used ina calibration process performed by the end user to measure and correctmisalignment between the two projectors whenever required, for example,after adjustment of the IPD, or as an automated self-calibration processperformed intermittently or, in certain preferred applications, wheneverthe device is powered-on.

Solving the relative orientation of cameras 44L and 44R (after IPDadjustment, as described for FIG. 3) is particularly straightforwardwhen the cameras are sampling images for a distant scene, since parallaxbetween the two sampled images is negligible. “Distant” in this contextwould ideally be any distance over about 100 meters, which ensures thatangular variations due to convergence between the eyes/cameras aresmaller than the angular resolution of human visual perception.Practically, however, “distant” here may include any distance over 30meters, and in some cases, distances of 10 or 20 meters may also allowuse of this simplified calibration process with acceptable results.Thus, in a case of user-actuated calibration, the user can be instructedto direct the device towards a distant scene before initiating thecalibration process. Similarly, where the device is used in an outdoorenvironment, the device may be configured to detect, either via aranging sensor or by image processing, when the cameras are viewing adistant scene. Calibration can then be formed by sampling images fromthe distant scene from each camera 44L and 44R, and performing imagecomparison/registration between the two images to determine atransformation between the cameras.

Straightforward image registration may sometimes be used for thealignment correction even where the scene is at short range, so long asthere is little “depth” to the scene and both cameras essentially samplethe same image. One such example would be calibration by imaging a flatsurface such as a poster or other picture or texture on a wall. In thiscase, information is needed regarding the distance from the cameras tothe surface, in order to correct for the convergence angle.

In order to allow calibration in a range of situations where “distantscenery” may not be available, or for a more robust calibration processsuitable for being performed automatically without user cooperation,calibration can also be performed using nearby objects, for whichparallax between the cameras is significant. In this case, a 3Dreconstruction is needed in order to ‘solve’ the relative camerapositions. Movement of the cameras may be needed to generate multipleimages for accurate solutions, as illustrated schematically in FIG. 7.Algorithms for this calculation are well known, for example, in theliterature and open-source code libraries relating to SLAM (simultaneouslocation and mapping) processing. By employing these algorithms, a 3Dreconstruction (or “model”) of at least part of the scene is generatedfor each camera. The offset of the reconstruction between the cameras isused to determine the offset (spatial and orientation) between theprojectors.

Where SLAM processing is used to derive a model, a scaling factor isneeded to fully resolve the model. This scaling factor may be derivedfrom any of a number of sources including, but not limited to: a knowndistance between the two cameras in the case of a device without IPDadjustment; a measured distance between the two cameras, where anencoder is included on the IPD adjustment mechanism; camera motion asderived from an inertial motion sensor arrangement integrated with thedevice; a distance to a pixel location within one of the images asderived, for example, by a rangefinder integrated with the device;identification of an object of known dimensions included within thefield of view of the images; and introduction of additional parameterconstraints such as, for example, objects known to have straight edgesor the like.

An exemplary overview of the total overall process in a case of IPDadjustment and subsequent realignment is shown in FIG. 8. First, theprocess here is assumed to be initiated after an adjustment of distancebetween the projectors, such as by an IPD adjustment (step 110), and maybe user-initiated or automatically triggered. The process can also beimplemented as an automatic or semi-automated process, performed onstart-up of the device or triggered manually or by a software triggersignal, optionally with prompts generated to prompt the user to moverelative to the viewed scene.

Once triggered, the device acquires images of the scene for the leftcamera (step 112) and the right camera (step 114), and the processingsystem (onboard the device, local or remote) compares the images toderive the relative orientations of the two cameras (step 116). Wherethe simple registration process fails due to parallax variations betweenthe images, the system preferably samples additional images and waitsfor motion if required (step 118) to derive an at least partial 3D modelof part of a field of view, thereby allowing derivation of the relativecamera orientations. At step 120, this relative camera orientation datais used together with the previously-derived left camera to leftprojector transformation data (122) and right camera to right projectortransformation data (124) to determine an overall alignment correctionfor each projector which is introduced into the corresponding firmware(steps 126 and 128), thereby allowing a left virtual image to beconverted to a left transformed virtual image for projection fromprojector 40L and a right virtual image to be converted to a righttransformed virtual image for projection from projector 40R, so as togenerate correctly aligned viewed images.

Turning now to a second subset of methods of alignment correction forthe right-eye and left-eye displays of a binocular augmented realitydisplay device, FIGS. 9A and 9B illustrate schematically an arrangementin which a user provides input to define at least part of the alignmentcorrection. Thus, in FIG. 9A, there is shown an optical device similarto that of FIGS. 3 and 4, but with addition of a user input device 130,which may be a joystick, a touchscreen or any other suitable user inputdevice, optionally implemented as an APP running on a mobile electronicdevice. As before, this approach assumes the presence of a left camera44L spatially associated with the left-eye augmented reality display(projector 40L and out-coupling optical element 42L), and correspondingelements (a right camera spatially associated with the right-eyeaugmented reality display) for the right-eye side of the device (notshown).

It is a particular feature of certain particularly preferredimplementations according to this aspect of the present invention thatthe alignment correction method includes a first cross-registrationprocess including:

-   -   i. obtaining at least one image of a scene sampled by the right        camera,    -   ii. displaying via the left-eye augmented reality display at        least one alignment feature derived from the at least one image        sampled by the right camera,    -   iii. receiving an input from the user indicative of an alignment        offset between the at least one alignment feature and a        corresponding directly-viewed feature of the scene, and    -   iv. correcting a position of display of the at least one        alignment feature according to the user input until the at least        one alignment feature is aligned with the corresponding        directly-viewed feature of the scene. This defines a        transformation represented schematically by arrow 78 in FIG. 9B.

Most preferably, the alignment process also includes the reversecross-registration process, namely:

-   -   i. obtaining at least one image of a scene sampled by the left        camera,    -   ii. displaying via the right-eye augmented reality display at        least one alignment feature derived from the at least one image        sampled by the left camera,    -   iii. receiving an input from the user indicative of an alignment        offset between the at least one alignment feature and a        corresponding directly-viewed feature of the scene, and    -   iv. correcting a position of display of the at least one        alignment feature according to the user input until the at least        one alignment feature is aligned with the corresponding        directly-viewed feature of the scene. This defines a        transformation represented schematically by arrow 76 in FIG. 9B.

The user inputs are then used to implement an alignment correction tothe augmented reality display device. Where each camera is rigidlymounted relative to the corresponding augmented reality display, as inthe examples described above, the alignment correction is implementedusing relative alignment data for the right camera relative to theright-eye augmented reality display (arrow 74) and relative alignmentdata for the left camera relative to the left-eye augmented realitydisplay (arrow 72). Such data may be made available through a factoryalignment process, such as was described above with reference to FIGS. 5and 6.

In a more general case, where transformations 72 and 74 are unknown, ormay vary due to non-rigid (e.g., adjustable) mounting of the left/rightdisplays relative to the cameras, transformations 72 and 74 may beobtained by at least one additional registration process to receive userinputs for correcting an alignment of at least one of the right-eyeaugmented reality display and the left-eye augmented reality displayrelative to the corresponding one of the right camera and the leftcamera. These registrations processes can be performed in essentiallythe same way as the cross-registration processes described herein.

If all four transformations 72, 74, 76 and 78 are determined, there issome redundancy of information, since any three of these transformationsare in principle sufficient to determine an overall calibration matrixbetween the two displays. In practice, such redundancy is used toadvantage to improve accuracy of the alignment correction.

During the alignment process, each projector is activated separately. Atypical sequence of operation according to this approach would proceedas follows:

-   -   1) The user is instructed to look at scenery objects located at        the same nominal distance (apparent distance) as the virtual        image. The process is most simply implemented using “distant”        objects, to avoid issues of parallax compensation, although the        parallax issues can also be corrected, as discussed below.    -   2) The processing system injects the image from the camera of        one eye onto the adjacent projector, so that the observer sees        same augmented and ‘real world’ overlapping. If the scene is not        a “distant” scene, parallax compensation is introduced to the        projected image, according to an estimated distance to the        scene. A shift mismatch (offset) 57 (FIG. 9C) exists if the        camera and projector axes (after parallax compensation) are not        accurate.    -   3) The observer controls manually the position and rotation of        the virtual image and moves the augmented reality image to        overlap the ‘real world’ image 57 (mapping 72).    -   4) This process is repeated for second eye to generate mapping        74. Thus far, the calibration achieved is between each camera        and its adjacent projector.    -   5) The processing system injects the image from the camera of        one eye (44L) onto the opposite projector (40R) and lets the        user align the image, to determine mapping 76. The same is        repeated for the opposite camera and projector to generate        mapping 78. Now the two projectors and both cameras orientations        are calibrated.

The image (alignment feature) projected for this alignment process maybe at least part of the sampled image. In this case, the user gets a“double-vision” effect of superimposed images which do not quite fit,and adjusts the alignment until they are properly superimposed.

Alternatively, the projected alignment feature image may include one ormore location marker derived from the sampled images by imageprocessing, and corresponding to a feature detected in the sampledimage. This may be an outline of an object, or a number of markersdesignating “corner” features in the image. In this case, the useraligns these location markers with the corresponding features in thereal-world view.

Where the above process is performed using a scene which is not adistant scene, an estimate of distance to the scene is needed in orderto perform parallax corrections based on a known distance between eachcamera and the corresponding EMB center. This distance may be input bythe user, or may be derived by the system from any combination ofavailable sensors and/or image processing, depending on details of theapplication, as is known in the art. Non-limiting examples of how thedistance may be derived include: employing a rangefinder sensor,performing SLAM processing on images to derive a 3D model (as furtherdetailed above), and sampling images containing an object with knowndimensions.

Many projectors include optics that project the virtual image to afinite distance. In this case the calibration is preferably performedwhile viewing a scene at a distance matching the apparent distance ofthe virtual image. For example, if the virtual image is focused to 2meters, the calibration should preferably also be performed on a sceneor object located at a distance of about two meters. The injected imagefrom the camera to the projector is shifted according to parallaxbetween the camera and the projector (relative distance is known) at thespecified distance and center of field.

It is important to note the alignment procedures described here areapplicable also if the two projector/camera pairs are combined rigidlyduring production process, i.e., without adjustable spacing for the IPD.In this case, transformations 72 and 74 are typically precalibrated, asdescribed above, and only transformations 76 and 78 are achieved throughuser input.

In all of the cases herein where reference is made to “stereoscopicalignment correction”, this is typically implemented through generatinga calibration matrix relating each eye to the real world, or defining arelationship between the eyes.

An alternative approach to performing the cross-alignment of projectorsfor binocular augmented reality can be achieved without reliance onoutwards-looking cameras (which may or may not be present in theproduct). Instead, this third subset of alignment correction techniquesemploys a camera, separate from the augmented reality display device, tosample images simultaneously from both the right-eye display and theleft-eye display, and then derives an alignment correction from theimage. An exemplary implementation of this alternative approach ispresented below.

In general terms, a method for deriving an alignment correction betweena right-eye display and a left-eye display of a binocular augmentedreality display device according to this aspect of the present inventionincludes the steps of:

-   a) positioning a camera having a field of view so that the camera    field of view includes simultaneously part of a projected image from    the left-eye display and part of a projected image from the    right-eye display;-   b) projecting via each of the right-eye display and left-eye display    at least part of a calibration image including at least one    right-field alignment feature and at least one left-field alignment    feature;-   c) employing the camera to sample an image;-   d) identifying within the image the right-field alignment feature    and the left-field alignment feature; and-   e) deriving from a position within the image of the right-field    alignment feature and the left-field alignment feature an alignment    correction between the right-eye display and the left-eye display of    the augmented reality display device.

One implementation of this approach is illustrated here schematically inFIG. 10A. It will be noted that some of the light projected by waveguide42 toward the observer eye is reflected forward (i.e., outwards from theuser), for example, by the external surface of the waveguide closest tothe eye. In the implementation illustrated here, it is this outwardlyreflected light that is detected by a camera 80 positioned on anopposite side from the viewing side of the augmented reality displaydevice 40L, 42L so that the camera captures an outwardly reflectedportion of image illumination from each of the right-eye display and theleft-eye display.

The system controller injects an image to projector 40 that illuminatesthe eye through waveguide 42 as shown by the solid line arrows. Some ofthe light is reflected in the opposite direction as shown by thedash-dot line arrows.

A camera on a portable device 80 receives at least part of the forwardreflected image and transmits the image to the system controller forprocessing. (The camera is here illustrated only schematically, and willclearly be oriented facing towards the projector and positioned tocapture part of the forward-reflected image illumination.) Theprocessing can optionally be performed in the portable device itself.

Although only part of the field is received by camera 80, the image isdesigned so that it is possible to derive what part of the image isreceived, as discussed further below with reference to FIG. 11D. Fromthat part, the processor derives the orientation of the camera relativeto the forward projected image.

FIG. 10B shows schematically two projectors 99L and 99R, each indicativeof the projector orientation for both eyes of the corresponding device.In 99L, the ray 100 is projected toward the observer perpendicularly tothe faces of waveguide 99L and reflection 102 is therefore reflected inthe opposite direction, along the optical axis. In contrast, inwaveguide 99R, an alternative geometry is shown in which the projectedimage optical axis indicated by output ray 104 is not perpendicular tothe surface of waveguide 99R, and the reflected ray 106 is therefore notopposite to 104. Therefore, a calibration matrix should be derived forthe offset of 106 relative to 104. This calibration matrix should bederived by comparing forward images (100 and 104) with reflected images(102 and 106) during projector production or as described below.

Image acquisition according to this approach is performed simultaneouslyfor both projectors, as shown schematically in the plan view in FIG.11A. The dot-dashed arrows represent the forward-reflected image. Camera80 receives different sections of the reflected images from the twoprojectors and derives the orientation to both fields. By comparingthese orientations, it is possible to derive the relative orientationbetween projectors and correct the alignment electronically, as above.

Improved accuracy of calibration is achieved if camera 80 is placedfurther from projectors 42. In the case of a hand-held camera, whichcannot conveniently be held so far from the device, imaging from alarger effective distance can be achieved by observing the projectorsthough a mirror 57, as illustrated in FIG. 11B. This mirror-basedgeometry also allows this calibration technique to be implemented usinga built-in forward looking camera of the augmented reality displaydevice itself, particularly in devices provided with a single centralforward-looking camera.

The orientation of camera 80 can be optimized by providing visualguidance cues to the user for correct positioning of the camera duringcalibration. For example, if camera 80 is a camera of a mobile deviceintegrated with a screen, such as a mobile phone, at least oneindication to a user may be displayed via the screen to assist incorrect positioning of the camera, as illustrated in FIG. 11C.Additionally, or alternatively, for any hand-held camera, at least oneindication can be displayed to a user via one or both of the augmentedreality displays to assist in correct positioning of the camera.

FIG. 11D shows an example of an image that can be projected by the twodisplays for the calibration process. Other arbitrary images can beused, and this one is presented here as a non-limiting example. Theimage has clear markings 90 a and 90 b, which serve respectively as aleft-field alignment feature and a right-field alignment feature. Theright- and left-field alignment features may be part of a contiguousgeometric pattern, or may be isolated features, and are preferablydistinguishable from each other. They preferably include features thatare easily identified and processed by image processing techniques toderive position and orientation. The image is projected aftercompensation for any geometrical distortions introduced by the projectoritself. It will be noted that only a part of the image is captured bycamera 80 from each separate projector. The camera is positioned sothat, in the case of a camera on the “outside” of the projector, thesampled image includes the left-field alignment feature viewed via theright-eye display and the right-field alignment feature viewed via theleft-eye display.

FIG. 11E shows schematically an image 100 received by camera 80. Thedistance from camera 80 to the glasses can be derived from parameters onthe glasses, for example the glasses size 82 in the image. In thewaveguides 42R and 42L the reflections of the projected image areapparent as 84R and 84L. The images in both reflections include themarkings 90 a and 90 b. By measuring the angular distance in the imagebetween the markings 86, and considering the parallax caused by theknown distance to the glasses, it is possible to know the actualmisalignment between the projectors 42R and 42L. Angular misalignmentcan also be derived as shown by the skew angle designated 88. Thisarchitecture also enables detection of eye position 60R and 60L. Thisfurther improves projection alignment by taking into considerationdistortions caused by eye position in the projector eye-box.

In an alternative set of implementations, camera 80 is positioned on theviewing side of the augmented reality display device, i.e., the sidefrom which the user looks through the display. In this case, the sampledimage includes the right-field alignment feature viewed via theright-eye display and the left-field alignment feature viewed via theleft-eye display. An example of this implementation is shown in FIG.11F.

It is important that the camera 80 be focused onto the projected image.If lenses are placed in front of projectors 42 then the virtual image 51will be generated at a finite apparent distance (the apparent focaldistance). This should be considered when deriving the parallaxintroduced to 84R and 84L.

In the example of FIG. 11F, the projector includes lenses so that image51 is projected as virtual images 62L (from 42L) and 62R (from 42R) atan apparent focal distance 61. These two images should be brought intoexact overlapping relation for optimal alignment. The image acquired bycamera 80 will be equivalent to 84L and 84R (described in FIG. 11E), andthe derivation of the offset between 62L and 62R will consider thedistance to the virtual image 61 (preset by the lenses) and to thecamera 63 (again derived for example by identifying a dimension of thedevice 82 in the image).

As mentioned, the distance of camera 80 from the display device can bedetermined by identifying features associated with the display device,such as a width dimension 82, within the image. Ideally, in order todetermine both the distance and orientation of the camera relative tothe display device, the processing system preferably identifies withinthe image features associated with the binocular augmented realitydisplay device sufficient to define at least three, and most preferablyfour, non-collinear (and for four, non-coplanar) fiducial points. Thefeatures may be any feature relating to the shape of the device, or anyreference pattern formed on a surface of the device. In cases where theprojected calibration image is rendered at a specific focal depth,features of the projected virtual image may also in some cases be usedas fiducial points. The fiducial points are then processed to determinea position of the camera relative to the fiducial points, and hence tothe projectors.

An exemplary non-limiting implementation of this process is described inFIG. 12. As in FIG. 8 above, the calibration may be necessitated bymisalignment introduced by IPD adjustment (step 140), although it is notlimited to such cases. At step 142, the calibration image or “fieldimage” is “injected” for display via both the right and left eyeprojectors, and camera 80 is used to sample an image containing a partof the illumination corresponding to the calibration image from each ofthe projectors, and preferably also imaging the projectors or otherfeatures of the display device itself (step 144).

At step 146, the features of the display device are processed todetermine the camera orientation relative to each projector. This thenprovides sufficient information to allow derivation of the relativealignment of the projectors from the parts of the calibration imageacquired via each display (step 148). Where camera 80 is used on theoutside of the display with outwardly-reflected illumination, and wherethe image projection axis is non-perpendicular to the surfaces of thewaveguide, premeasured reflections offset parameters (150) are alsoemployed in the alignment calculation. The alignment calculations arethen used to generate calibration matrices for updating the firmware ofeach projector (step 152).

The camera on the portable device 80, can also be used to assist theuser during a mechanical IPD adjustment itself (before performing thecalibration described). According to this option, the user changes thedistance between the projectors while the camera transmits continuouslythe facet image to the processor. The processor compares the eyeposition to the optical projector position (which may optionally havemarkings on it to facilitate detection of the projector position), andgenerates an output to the user (typically an audio signal and/or avisual display) to indicate how the relative position should be furtheradjusted, or to inform the user when an optimal position has beenreached for the user. The calibration process is then preferablyperformed, as described herein.

It will be appreciated that the above descriptions are intended only toserve as examples, and that many other embodiments are possible withinthe scope of the present invention as defined in the appended claims.

What is claimed is:
 1. A binocular augmented reality system with alignment correction comprising: (a) a right-eye display unit comprising a first augmented reality display spatially associated with a first camera, said first camera is a forward-looking camera rigidly associated with said first augmented reality display; (b) a left-eye display unit comprising a second augmented reality display spatially associated with a second camera, said second camera is a forward-looking camera rigidly associated with said second augmented reality display; (c) an adjustable support structure providing an adjustable inter-pupil distance (IPD) between said right-eye display unit and said left-side display unit; and (d) a processing system comprising at least one processor, said processing system in data communication with said first and second cameras and with said first and second augmented reality displays, said processing system being configured to determine an alignment correction between said first and second augmented reality displays for binocular display of images, said alignment correction comprising: (a) performing a first cross-registration process comprising: (i) obtaining at least one image of a scene sampled by said first camera, (ii) displaying via said second augmented reality display at least one alignment feature derived from said at least one image sampled by said first camera, (iii) receiving an input from the user indicative of an alignment offset between the at least one alignment feature and a corresponding directly-viewed feature of the scene, and (iv) correcting a position of display of the at least one alignment feature according to the user input until the at least one alignment feature is aligned with the corresponding directly-viewed feature of the scene; (b) performing a second cross-registration process comprising: (i) obtaining at least one image of a scene sampled by said second camera, (ii) displaying via said first augmented reality display at least one alignment feature derived from said at least one image sampled by said second camera, (iii) receiving an input from the user indicative of an alignment offset between the at least one alignment feature and a corresponding directly-viewed feature of the scene, and (iv) correcting a position of display of the at least one alignment feature according to the user input until the at least one alignment feature is aligned with the corresponding directly-viewed feature of the scene; and (c) deriving said alignment correction based on said user inputs.
 2. The system of claim 1, wherein said processing system is responsive to a user input to determine said alignment correction.
 3. The system of claim 1, wherein said processing system is configured to determine said alignment correction intermittently.
 4. The system of claim 1, wherein said processing system is configured to determine said alignment correction while a user is looking through said first and second augmented reality displays.
 5. The system of claim 1, wherein said right-eye display unit, said left-eye display unit and said adjustable support are part of an augmented reality device, and wherein at least part of said processing system is an onboard processing system integrated with said device.
 6. The system of claim 1, wherein said right-eye display unit, said left-eye display unit and said adjustable support are part of an augmented reality device, and wherein at least part of said processing system is a remote processing system associated with said device via a short-range or long-range communication connection.
 7. The system of claim 1, wherein said processing system is configured to determine a correlation between images from said first camera and said second camera as part of determining said alignment correction.
 8. The system of claim 1, wherein said processing system is configured to determine said alignment correction by: (i) sampling at least one image from said first camera; (ii) sampling at least one image from said second camera; (iii) co-processing said images from said first and second cameras to derive an inter-camera mapping indicative of a relative orientation between said first camera and said second camera; and (iv) combining said inter-camera mapping with a first alignment mapping between said first camera and said first augmented reality display and a second alignment mapping between said second camera and said second augmented reality display to derive said alignment correction.
 9. The system of claim 8, wherein said at least one image sampled from said first camera and from said second camera are multiple images, and wherein said co-processing includes deriving a three-dimensional model of at least part of a scene included in said multiple images.
 10. The system of claim 1, wherein said processing system is configured to receive a user input indicative of an alignment adjustment between an alignment feature displayed via said first augmented reality display and a corresponding real-world feature observed by a user, said alignment feature being derived from an image sampled from said second camera.
 11. The system of claim 10, wherein said processing system is further configured to receive a user input indicative of an alignment adjustment between an alignment feature displayed via said second augmented reality display and a corresponding real-world feature observed by a user, said alignment feature being derived from an image sampled from said first camera.
 12. The system of claim 1, wherein said at least one alignment feature for each of said cross-registration processes is at least part of the sampled image.
 13. The system of claim 1, wherein said at least one alignment feature for each of said cross-registration processes is a location marker corresponding to a feature detected in the sampled image.
 14. The system of claim 1, further comprising obtaining an estimated distance to an object in the sampled image, said estimated distance being employed to implement said alignment correction. 